The wide adoption of the Internet, and networked computing in general has resulted in the proliferation of networked computing systems. Networked computing systems may consist of various computers or other devices that communicate over a network. For computer networking a client server terminology is often used. A computer that makes a request over a network is referred to as a client and a computer that receives such a request and replies to it is referred to as a server. Since networked computing systems may be very complex and may be used for mission critical tasks, testing of these systems is very important. Furthermore, these systems may need to be monitored. Monitoring can have many uses such as auditing, identifying errors, or monitoring the behavior of clients who are using a server.
The testing, auditing and monitoring tasks may be accomplished by utilizing features of the software that runs on the computers of these systems. Most networking software, especially server software, includes certain testing, monitoring and auditing capabilities. But, utilizing these features may change the environment being tested or monitored. For example, if a server's speed of replying to certain requests is being monitored utilizing a monitoring feature of the server, the monitoring feature itself will use processor time and thus affect the speed with which the processor replies to requests. If software is being monitored to discover errors, the monitoring feature of the software adds an extra level of complexity which can itself be the cause of errors. The tendency of a monitoring feature to change the environment being monitored is called intrusiveness. Intrusive monitoring features usually pose an increasing risk of destabilizing the system or skewing the results of monitoring operations.
Encryption is often used in network communications. Monitoring encrypted information can be difficult, especially if such monitoring is to be performed in a non-intrusive fashion. For monitoring to be effective the information must be decrypted.
In order for monitoring to be effective for some purposes, it needs to be performed in real time, i.e., within a relatively short time of the communications being monitored. For example, for some server installations it is advantageous for errors to be detected and handled shortly after they occur. Thus, it would be advantageous to have an alert or alarm that goes off as soon as an error occurs. Similarly, alarms are useful for other kinds of events, such as degradations of performance. Furthermore, it may be advantageous for a server operator, or a support professional to know at any given time the way clients are currently using the server. If the systems that are being monitored are sensitive to downtime, it is desired that diagnostic information is obtained very soon after any errors (or other performance related events) are detected, so they can be addressed with minimal degradation to performance. Real time monitoring also allows for early discarding of unnecessary data. Some non-real time monitoring applications create large dump files of raw monitoring data that contain mostly unnecessary data and take up valuable system resources.
The communications between a server and a client may be quite complex. In particular any single communication may refer to or depend on information from other communications. Thus, it is important for monitoring purposes that related communications are examined together and their relationships noted. For example, in the case of communications in HTTP, an HTML page may refer to various picture files or frame files that have been sent at a previous time, or which must be sent at a later time.
Modern computer networking is usually described by the OSI seven layer model which is well known in the art. According to the model, data sent over a network may change its form as it passes through the utilities that service each layer. For example, at the network layer data is split up into packets and header information is added to each packet. As it is passed through a network cable the data is considered to be in layer 1 form. However, for many monitoring and testing purposes, other layers are more relevant. For example, when monitoring is done from the end-user's perspective, layer 7 is most relevant, layer 7 information is closest in form to the information entered or viewed by the end-user.
There are several known types of monitoring applications. One of them is the proxy. Proxies are usually placed on a communication channel between two or more monitored machines and actively forward messages between these machines. Proxies are intrusive, because in order to use proxies the monitored machines must be configured to address their communications to the proxy. Another intrusive feature of proxies is that they add a point of failure, that is a failure of the proxy will affect the communications going through it. In fact a failure of the proxy usually means no communications can go through it. Proxies also cause delays in communications.
Another known type of monitoring application is a server side plug-in. The server side plug-in is a software function that interacts with a server by way of an interface of the server, created specifically for server side plug-ins. Examples of such interfaces for web servers are ISAPI, NSAPI and CGI. Occasionally, the server side plug-in is directly inserted in the server software and executed by the server when predefined conditions occur. Different servers have different plug-in interfaces, thus a server plug-in will have to be specifically configured for use with one or more types of servers. Server plug-ins usually have access to OSI level 7 data (which may include the decryption of encrypted communications). But, server plug-ins change the behavior of the server by actively interacting with it. Thus, they are intrusive. The intrusive nature of server plug-ins causes performance degradation and adds a point of failure (a failed server plug-in may cause failure or performance problems with the server).
Another known type of monitoring application is the use of log files. Some communications applications, such as web servers, may be configured to continuously add data about their operation to log files. For web servers, such data is usually the URL's of web pages that have been served. Log files require storage space and applications use up additional processing resources when they add information to log files. Because the size of log files is limited by practical considerations, only limited amount of information is stored in log files. For example, message bodies are usually not stored in log files. Log files are also intrusive, because an application needs to be configured to use a log file. Thus, log files can degrade application performance, or even cause failures.
Network sniffers are non-intrusive type of monitoring devices. But, they do not have access to encrypted information. They do not possess the ability to reconstruct data to the application layer (OSI layer 7). Network sniffers usually do not process data in real time, but store data for later processing. Some network sniffers, such as intrusion detectors, may process data in real time in order to detect certain events.